Progressive multi-level distillation learning for pruning network

Abstract Although the classification method based on deep neural network has achieved excellent results in tasks, it is difficult to apply real-time scenarios because of high memory footprints and prohibitive inference times. Compared unstructured pruning, structured pruning techniques can reduce computation cost model runtime more effectively, but inevitably reduces precision model. Traditional methods use fine tuning restore damage performance. However, there still a large gap between pruned original one. In this paper, we progressive multi-level distillation learning compensate for loss caused by pruning. Pre-pruning post-pruning networks serve as teacher student networks. The proposed approach utilizes complementary properties knowledge distillation, which allows learn intermediate output representations network, thus reducing influence subject Experiments demonstrate that our performs better CIFAR-10, CIFAR-100, Tiny-ImageNet datasets with different rates. For instance, GoogLeNet achieve near lossless CIFAR-10 dataset 60% Moreover, paper also proves using during process achieves significant performance gains than after completing